INFERENCE COSTS

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts (2025.findings-acl)

Challenge:	Large language models are often inefficient for real-world deployment due to expensive inference costs.
Approach:	They propose to use knowledge distillation to transfer the knowledge of the original model to a smaller, more efficient student model.
Outcome:	The proposed method is the best for multi-lingual and multilingual student architectures.

Copied to clipboard

Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, Jianfeng Gao

Challenge:	Large Language Models (LLMs) have outstanding performance by learning a large number of model parameters on large amounts of data.
Approach:	They propose a method of grouping and pruning similar experts to improve the model’s parameter efficiency by a range of natural language tasks.
Outcome:	The proposed method outperforms other model pruning methods on a range of natural language tasks.

Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model (2022.naacl-industry)

Copied to clipboard

Gongzheng Li, Yadong Xi, Jingzhen Ding, Duan Wang, Ziyang Luo, Rongsheng Zhang, Bai Liu, Changjie Fan, Xiaoxi Mao, Zeng Zhao

Task-Level Instructions Induction for Audio Question Answering from Few Examples (2026.eacl-short)

Challenge:	Recent studies show that transformer-based models are effective over many tasks, but they are expensive to deploy in the industrial application.
Approach:	They propose a transformer-based inference solution that optimizes kernels for long inputs and large hidden sizes and a flexible CUDA memory manager to reduce the memory footprint when deploying a large model.
Outcome:	The proposed solution achieves an average speedup of 1.40-4.20x on the transformer decoder layer with an A100 GPU.

Copied to clipboard

Po-Chun Chen, Hen-Hsen Huang, Hsin-Hsi Chen

Generating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling (2025.naacl-srw)

Challenge:	Large audio-language models benefit from Chain-of-Thought (CoT) prompting for audio question answering (AQA) however, acquiring audio CoT examples is difficult as it requires sequential listening and careful integration of acoustic and linguistic information.
Approach:	They propose a method which induces reusable task instructions from few audio examples once per task.
Outcome:	Evaluated on 9 LALMs across two benchmarks, Audio-Induct outperforms state-of-the-art prompting methods while maintaining low inference costs.

Copied to clipboard

Samuel Belkadi, Libo Ren, Nicolo Micheletti, Lifeng Han, Goran Nenadic

Your Reasoning Model Knows What Counts: Self-Guided Chain-of-Thought Pruning for Efficient Reasoning (2026.acl-long)

Challenge:	Existing methods to generate medical records using Causal Language Modelling are limited due to privacy concerns.
Approach:	They propose a method for generating medical records using Masked Language Modelling using Causal language models.
Outcome:	The proposed method produces high-quality synthetic data with a re-identification risk of only 3.5% and a patient recall of 96%.

Copied to clipboard

Zi-Ao Ma, Xian-Ling Mao, Tian Lan, Chen Xu, Zhijing Wu

Progressive Fine-Tuning for Cost-Effective Structured Attribute Generation in E-commerce (2026.acl-industry)

Challenge:	Existing approaches to Chain-of-Thought reasoning are often degraded because they disregard the model’s intrinsic reasoning dependency.
Approach:	They propose a self-guided pruning framework that leverages the model’s intrinsic likelihood landscape to identify segments that are extraneous to its specific reasoning pattern.
Outcome:	The proposed framework reduces output length while maintaining or improving accuracy on multiple benchmarks.

Copied to clipboard

Lakshman Kolasani, Fatemeh Taheri Dezaki

Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks (2022.findings-emnlp)

Challenge:	Large language models excel at structured information generation but face cost and latency challenges when deployed at scale in user-facing products.
Approach:	They propose a parameter efficient supervised fine-tuning pipeline for adapting a small language model to structured attribute generation in e-commerce product listing.
Outcome:	The proposed model reduces inference costs by 98% and latency by 70% on a large-scale product listing service while preserving an 86.4% user acceptance rate.

Copied to clipboard

Sishuo Chen, Wenkai Yang, Zhiyuan Zhang, Xiaohan Bi, Xu Sun

Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia: Quality vs. Energy Consumption (2026.eacl-srw)

Challenge:	Existing online backdoor defense methods for NLP models focus on anomalies at input or output level, causing fragility to adaptive attacks and high computational cost.
Approach:	They propose a feature-based online defense method to detect poisoned samples . they use a distance-based anomaly score to distinguish poisones from clean samples based on feature-level regularization .
Outcome:	The proposed method outperforms existing methods in sentiment analysis and offense detection tasks.

Copied to clipboard

Patrycja Smits, Tomasz Walkowiak

Batch Prompting: Efficient Inference with Large Language Model APIs (2023.emnlp-industry)

Challenge:	Retrieval-augmented generation systems are a dominant paradigm for knowledge-intensive applications.
Approach:	They evaluate language models from 4B to 70B parameters within a Polish Wikipedia-based RAG pipeline.
Outcome:	The proposed model selection process reduces energy consumption by 83% and improves quality.

Copied to clipboard

Zhoujun Cheng, Jungo Kasai, Tao Yu

Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching (2025.acl-short)

Challenge:	Performing inference on large volumes of samples can be computationally and financially costly.
Approach:	They propose a prompting approach that enables large language models to run inference in batches instead of one sample at a time.
Outcome:	The proposed prompting reduces both token and time costs while retaining downstream performance.

Copied to clipboard

Juan Wisznia, Cecilia Bolaños, Juan Tollo, Giovanni Franco Gabriel Marraffini, Agustín Andrés Gianolini, Noe Fabian Hsueh, Luciano Del Corro

Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization (2024.emnlp-industry)

Challenge:	a new framework for analyzing sorting algorithms in pairwise ranking prompting (PRP) is developed to re-center the cost model around LLM inferences rather than traditional pairwise comparisons.
Approach:	They propose a framework for analyzing sorting algorithms in pairwise ranking prompting (PRP) they propose to re-center the cost model around LLM inferences rather than traditional pairwise comparisons.
Outcome:	The proposed framework encourages strategies such as batching and caching to mitigate inference costs.

Copied to clipboard

Md Tahmid Rahman Laskar, Elena Khasanova, Xue-Yong Fu, Cheng Chen, Shashi Tn

L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models (2025.acl-long)

Challenge:	Existing LLMs require a new call to the inference endpoint/API for each new query . repeated calls to the endpoints/AP Is expensive and impractical for many real-world use cases.
Approach:	They compare the performance of various LLMs for query-based meeting summarization . they find that combining queries for the same context in a single prompt can be used to minimize repeated calls.
Outcome:	The proposed approach reduces the number of calls to the inference endpoints/APIs in meeting summarization tasks.

Copied to clipboard

Hyesung Jeon, Yulhwa Kim, Jae-Joon Kim

EXPLAIN: Enhancing Retrieval-Augmented Generation with Entity Summary (2025.acl-industry)

Challenge:	Quantization-aware PEFT methods have been developed to reduce memory and computational costs associated with large language models.
Approach:	They propose a method that integrates Quantization-Aware Training (QAT) with LoRA to reduce memory overhead and improve model accuracy.
Outcome:	The proposed method significantly reduces QAT’s memory overhead while preserving the advantage of QAT in producing fully quantized LLMs with high accuracy.

Copied to clipboard

Yaozhen Liang, Xiao Liu, Jiajun Yu, Zhouhua Fang, Qunsheng Zou, Linghan Zheng, Yong Li, Zhiwei Liu, Haishuai Wang

Explainability & Interpretability in NLP

Challenge:	Existing document question answering methods reduce inference costs and input tokens.
Approach:	They propose a retrieval-augmented generation method that automatically extracts useful entities and generates summaries from documents.
Outcome:	The proposed method surpasses baseline retrieval-augmented generation (RAG) and long-context question answering (LC) methods achieve higher accuracy by processing entire documents, but at the cost of increased computational Corresponding authors.

In-Context Former: Lightning-fast Compressing Context for Large Language Model (2024.findings-emnlp)

Copied to clipboard

Xiangfeng Wang, Zaiyi Chen, Tong Xu, Zheyong Xie, Yongyi He, Enhong Chen

Confidence-Aware Reasoning: Optimizing Self-Guided Thinking Trajectories in Large Reasoning Models (2025.emnlp-industry)

Challenge:	Existing methods to reduce inference costs of transformer-based large language models entail quadratic complexity . et al., 2017): transformer-derived large language model performance is a major challenge.
Approach:	They propose a method that compresses long contexts into short soft prompts . they use the self-attention mechanism of the large model to extract and condense information .
Outcome:	The proposed method reduces compression costs by 68 to 112 times while achieving 90% of baseline performance.

Copied to clipboard

Jiaxin Zhang

Visual Program Distillation with Template-Based Augmentation (2025.findings-emnlp)

Challenge:	Chain-of-thought enables large reasoning models to reason through multi-step problems but often leads to unnecessarily long or redundant reasoning traces, a phenomenon known as overthinking.
Approach:	They propose an inference-time framework that selectively prunes low-utility reasoning blocks and halts early when sufficient confidence has been achieved.
Outcome:	The proposed framework improves answer accuracy by up to +13.3% while reducing average reasoning length by 40%–50%.

Copied to clipboard

Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem

LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation (2025.emnlp-main)

Challenge:	Adapting visual programming to specialized tasks or domains remains challenging due to high annotation and inference costs.
Approach:	They propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations.
Outcome:	The proposed method can generate high-quality visual programs with no human-generated annotations with a relatively small amount of question/answer data.

Copied to clipboard

Keisuke Kamahori, Jungo Kasai, Noriyuki Kojima, Baris Kasikci

Hierarchical Document Refinement for Long-context Retrieval-augmented Generation (2025.acl-long)

Challenge:	Modern automatic speech recognition systems rely on encoder-decoder architectures and their encoders are a critical bottleneck for efficient deployment due to high computational intensity.
Approach:	They propose a low-rank compression scheme for ASR encoders that leverages the strong low-ranked properties observed in intermediate activations and approximates linear transformations with a chain of low-Rank matrix multiplications.
Outcome:	The proposed method reduces inference costs while maintaining transcription accuracy while preserving low-rank properties observed in intermediate activations.

Copied to clipboard

Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Ye Qi, Zhicheng Dou

Information Retrieval

Challenge:	Real-world RAG applications often encounter long-context input scenarios where redundant information and noise results in higher inference costs and reduced performance.
Approach:	They propose an efficient plug-and-play refiner that leverages the structural characteristics of long documents.
Outcome:	Experiments on seven QA datasets show that LongRefiner achieves competitive performance in various scenarios while using 10x fewer computational costs and latency compared to baseline.

Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism (2024.findings-acl)

Copied to clipboard

Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai

AdaptThink: Reasoning Models Can Learn When to Think (2025.emnlp-main)

Challenge:	Existing approaches to generate draft tokens in large language models are expensive and resource-intensive.
Approach:	They propose an approach to generate draft tokens using a segment of the LLM and a self-distillation method to enhance the quality of draft token.
Outcome:	The proposed approach generates draft tokens using a segment of the LLM and a self-distillation method to improve quality and speed up generation.

Copied to clipboard

Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, Juanzi Li

QUIK: Towards End-to-end 4-Bit Inference on Generative Large Language Models (2024.emnlp-main)

Challenge:	Recent advances in large reasoning models have demonstrated remarkable capabilities in tackling complex tasks.
Approach:	They propose an algorithm to teach reasoning models to choose the optimal thinking mode based on problem difficulty.
Outcome:	The proposed algorithm reduces the average response length and improves accuracy on three math datasets.

Copied to clipboard

Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, Dan Alistarh

SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models (2025.findings-naacl)

Challenge:	Large Language Models (LLMs) are extremely popular, leading to a race towards reducing their inference costs.
Approach:	They propose a method that quantizes weights and activations to 4 bits to achieve better accuracy.
Outcome:	The proposed method reduces runtime costs in memory-bound models but does not address cost-bound scenarios.

Copied to clipboard

Jahyun Koo, Yerin Hwang, Yongil Kim, Taegwan Kang, Hyunkyung Bae, Kyomin Jung

SKIntern: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models (2025.coling-main)

Challenge:	Knowledge Distillation (KD) has emerged as a popular method for compressing large language models due to high inference costs and memory requirements.
Approach:	They propose a method that integrates the teacher model during the student's sequence generation to reduce misguidance from the teacher.
Outcome:	Experiments on three model families and five instruction-following datasets show that SWITCH surpasses traditional methods, especially in the generation of long sequential data.

Copied to clipboard

Huanxuan Liao, Shizhu He, Yupu Hao, Xiang Li, Yuanzhe Zhang, Jun Zhao, Kang Liu

Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis (2026.acl-long)

Challenge:	Large Language Models (LLMs) have high computational costs and privacy concerns due to their high computational expenses and data privacy.
Approach:	They propose a method that empowers SLMs to internalize symbolic knowledge and few-shot examples gradually through a progressive fine-tuning process.
Outcome:	The proposed approach outperforms state-of-the-art baselines by over 5% while reducing inference costs by up to 4 across a wide range of SLMs in both in-domain (ID) and out-of domain (OOD) tasks.

Copied to clipboard

Zehua Pei, Hui-Ling Zhen, Lancheng Zou, Xianzhi Yu, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu

Adapting Language Models to Compress Contexts (2023.emnlp-main)

Challenge:	Large language models (LLMs) are fast but require expensive pre-training . a new approach to scale large language models into MoEs reduces inference costs .
Approach:	They propose an analytical post-training framework that rapidly restructures FFNs into sparse MoE architectures using only a small calibration dataset.
Outcome:	The proposed framework outperforms existing methods on a small calibration dataset.

Copied to clipboard

Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen

Large Language Model as a Teacher for Zero-shot Tagging at Extreme Scales (2025.coling-main)

Challenge:	Transformer-based language models have a finite context window and expensive computational cost of processing long text documents.
Approach:	They propose to adapt pre-trained LMs into AutoCompressors to compress text into summary vectors . authors propose to use summary vector to speed up inference over long contexts based on a finite context window .
Outcome:	The proposed model can compress long contexts into summary vectors, which are accessible as soft prompts.

Copied to clipboard

Jinbin Zhang, Nasib Ullah, Rohit Babbar

Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack (2025.findings-acl)

Challenge:	Extreme Zero-shot XMC uses lightweight bi-encoders to identify pseudo labels . state-of-the-art methods rely on suboptimal labels for training .
Approach:	They propose a framework that uses a lightweight bi-encoder to identify high-quality pseudo labels during training while employing a lightweight bi-coder for efficient inference.
Outcome:	The proposed framework achieves superior performance and efficiency over existing methods.

Copied to clipboard

Murong Yue, Ziyu Yao

CoT-Valve: Length-Compressible Chain-of-Thought Tuning (2025.acl-long)

Challenge:	a recent study reveals a significant security vulnerability in batch prompting . malicious users can inject attack instructions into a batch, leading to unwanted interference .
Approach:	They construct a batch prompting benchmark and test it against other LLMs to find out if batch prompts are vulnerable.
Outcome:	The proposed approach achieves 95% accuracy in detecting attacks.

Copied to clipboard

Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, Xinchao Wang

DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding (2026.eacl-long)

Challenge:	Wei et al., 2022) have developed a powerful method for enhancing the reasoning capabilities of large language models.
Approach:	They propose to use a tuning and inference strategy to control the length of reasoning chains by a parameter space direction to control their length.
Outcome:	The proposed method reduces reasoning chains on GSM8K from 741 to 225 tokens with a minor performance drop (95.07% to 94.92%) and on AIME from 6827 to 4629 tokens, with only one additional incorrect answer.

Copied to clipboard

Moulik Choraria, Xinbo Wu, Akhil Bhimaraju, Nitesh Sekhar, Yue Wu, Xu Zhang, Prateek Singhal, Lav R. Varshney

PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning (2024.findings-emnlp)

Challenge:	Recent work shows that hyperscaling of data and parameter count in LLMs is yielding diminishing improvement when weighed against training costs.
Approach:	They propose to insert multimodal tokens directly into the middle of the model to bypass the early layers.
Outcome:	The proposed method reduces training and inference costs while preserving performance.

Copied to clipboard

Gyeongman Kim, Doohyuk Jang, Eunho Yang

Prompt Compression for Large Language Models: A Survey (2025.naacl-long)

Challenge:	Recent advances in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression.
Approach:	They propose a method that utilizes prompt tuning to enable generative language models to transfer student-friendly knowledge.
Outcome:	Extensive experiments on instruction-following datasets show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher’s parameters as prompts.

Copied to clipboard

Zongqian Li, Yinhong Liu, Yixuan Su, Nigel Collier

Efficiently Editing Mixture-of-Experts Models with Compressed Experts (2025.findings-emnlp)

Challenge:	Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input.
Approach:	They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs.
Outcome:	The proposed methods are categorized into hard prompt methods and soft prompt methods.

Copied to clipboard

Yifei He, Yang Liu, Chen Liang, Hany Hassan Awadalla

LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)

Challenge:	Mixture-of-Experts models allow for efficient scaling of large language models . fewer experts reduce computational costs, while more experts improve performance .
Approach:	They propose to activate only a subset of experts during training and inference . they propose compressed experts that preserve the most important experts .
Outcome:	The proposed approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts.

Copied to clipboard

Jung Hyun Lee, Jeonghoon Kim, June Yong Yang, Se Jung Kwon, Eunho Yang, Kang Min Yoo, Dongsoo Lee

Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data (2026.findings-acl)

Challenge:	Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding.
Approach:	They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices.
Outcome:	The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs.

Copied to clipboard

Xuanming Zhang, Shwan Ashrafi, Aziza Mirsaidova, Amir H. Rezaeian, Miguel Ballesteros, Lydia Chilton, Zhou Yu, Dan Roth

A Two-Agent Game for Zero-shot Relation Triplet Extraction (2024.findings-acl)

Challenge:	Recent work has explored reasoning efficiency via test-time scaling and early exit strategies.
Approach:	They propose an anytime reasoning framework and the Anytime Index to improve model quality . they also propose an inference-time self-improvement method to produce better intermediate solutions .
Outcome:	The proposed method improves on NaturalPlan, AIME, and GPQA datasets and improves reasoning quality and efficiency under budget constraints.

Copied to clipboard

Ting Xu, Haiqin Yang, Fei Zhao, Zhen Wu, Xinyu Dai

Low-Resource NLP

Challenge:	Existing methods for relation triplet extraction rely on labeled data and are limited in their applicability.
Approach:	They propose a two-agent game approach to deliberate and debate unseen relations by two agents, a generator and an extractor.
Outcome:	The proposed method outperforms baseline methods by 6%-16% in F1 scores.

Byte Latent Transformer: Patches Scale Better Than Tokens (2025.acl-long)

Copied to clipboard

Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srini Iyer

NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization (2024.findings-emnlp)

Challenge:	Existing large language models (LLMs) are trained on bytes, except for tokenization, which groups bytes into a static set of tokens.
Approach:	They propose a new byte-level LLM architecture that encodes bytes into dynamically sized patches, which serve as the primary units of computation.
Outcome:	The proposed architecture matches tokenization-based models with improvements in inference efficiency and robustness.

Copied to clipboard

Duy-Tung Pham, Thien Trang Nguyen Vu, Tung Nguyen, Linh Ngo, Duc Nguyen, Thien Nguyen

Information Extraction & Text Mining

Challenge:	Recent advances in neural topic models have focused on two main directions: the integration of the inference network with a pre-trained language model and the modeling of the relationship between words and topics in the generative model.
Approach:	They propose a framework to maximize mutual information between topics and PLMs . Experimental results indicate that NeuroMax reduces inference time .
Outcome:	The proposed framework reduces inference time, generates more coherent topics and topic groups, and produces more representative document embeddings.

Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM (2024.emnlp-main)

Copied to clipboard

Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, Tagyoung Chung

CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing (2022.acl-long)

Challenge:	Contrastive decoding (CD) improves the next-token distribution of a large expert language model (LM) using a small amateur LM.
Approach:	They propose a new unsupervised decoding method called Asymptotic Probability Decoding (APD) that extrapolates the probability curves from the LMs of different sizes to infer the asymptototic probabilities from an infinitely large LM.
Outcome:	The proposed method improves the next-token distribution of a large expert language model using a small amateur LM.

Copied to clipboard

Chen Liang, Pengcheng He, Yelong Shen, Weizhu Chen, Tuo Zhao

CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)

Challenge:	Existing work has resorted to sharing weights among models, but results are not affordable for real-world deployment.
Approach:	They propose a consistency-regularized ensemble learning approach based on perturbed models to retain ensemble benefits while maintaining a low memory cost.
Outcome:	The proposed approach outperforms the standard ensemble of 8 BERT-base models on the GLUE benchmark by 0.7 with a significantly smaller model size.

Copied to clipboard

Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding

When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2023.acl-long)

Challenge:	Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks.
Approach:	They propose a mix-of-experts model that allows the model size to grow without raising training costs.
Outcome:	The proposed model outperforms existing models in perplexity and robustness tests.

Copied to clipboard

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi

Multi-pass Decoding for Grammatical Error Correction (2024.emnlp-main)

Challenge:	Large language models struggle with tasks requiring rich world knowledge, implying the difficulty of encoding a wealth of world knowledge in their parameters.
Approach:	They propose a retrieval-augmentation method that improves performance and reduces inference costs by only retrieving non-parametric memories when necessary.
Outcome:	The proposed method improves performance and reduces inference costs by only retrieving non-parametric memories when necessary.

Copied to clipboard

Xiaoying Wang, Lingling Mu, Jingyi Zhang, Hongfei Xu

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning (2025.coling-main)

Challenge:	Seq2edit models decode only once without aware of subsequent tokens.
Approach:	They propose to iteratively refine the correction results of seq2seq models via Multi-Pass Decoding (MPD) to improve performance, but MPD increases inference costs . they propose to merge the source input and previous round correction result into one sequence.
Outcome:	Experiments on the CoNLL-14 and BEA-19 test set show that the proposed approach improves over baselines.

Copied to clipboard

Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang

SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context (2025.findings-emnlp)

Challenge:	Low-rank adaptation and its variants have been popular due to their ability to avoid excessive inference costs.
Approach:	They propose a low-rank adaptation method that enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight.
Outcome:	The proposed method outperforms LoRA and other fine-tuning methods across tasks with less trainable parameters.

Copied to clipboard

Hairu Wang, Yuan Feng, Yukun Cao, Xike Xie, S Kevin Zhou

Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models (2023.acl-long)

Challenge:	Large language models incur high inference costs during deployment, causing hallucination . no dedicated routing methods exist for RAG, and existing training-based routers face challenges scaling to this domain .
Approach:	They propose a plug-and-play routing framework that optimizes performance and cost . the framework delivers over 3x higher routing effectiveness while reducing runtime to less than 0.001x .
Outcome:	The proposed framework delivers over 3x higher routing effectiveness while reducing runtime to less than 0.001x compared to existing methods.

Copied to clipboard

Junmo Kang, Wei Xu, Alan Ritter

Multilingual Lottery Tickets to Pretrain Language Models (2023.findings-emnlp)

Challenge:	Increasing the size of pre-trained models can consistently improve performance on downstream tasks after fine-tuning, as seen in studies based on BERT, RoBERTa, T5 and empirical scaling laws.
Approach:	They propose to use knowledge distillation to build a compact model with a fixed budget instead of annotating data and manually labeling it.
Outcome:	The proposed approach reduces inference costs by reducing costs by hiring annotators and labelling data.

Copied to clipboard

Jaeseong Lee, Seung-won Hwang

Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models (2025.naacl-long)

Challenge:	Multilingual pretrained language models (mPLMs) have become the de-facto standard for multilingual tasks.
Approach:	They propose to search for a per-language subnetwork with comparable performance to the full model by scaling the model to reduce interference and then redistributing parameters to keep the parameters reduced.
Outcome:	The proposed model reduces the inference cost of models for each language while keeping the capacity per language more or less the same.

Copied to clipboard

Soham Poddar, Paramita Koley, Janardan Misra, Niloy Ganguly, Saptarshi Ghosh

Improved Sparse Upcycling for Instruction Tuning (2025.coling-main)

Challenge:	Large language models (LLMs) are recognized for their exceptional generative capabilities and versatility across various tasks.
Approach:	They conduct a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks to determine the impact of different models, tasks, prompts, and system-related factors on inference.
Outcome:	The proposed model energy benchmarks show that quantization and optimal batch sizes can significantly reduce energy usage.

Copied to clipboard

Wangyi Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning (2025.acl-long)

Challenge:	Existing methods for sparse upcycling lead to performance degradation in instruction tuning scenarios.
Approach:	They propose a representation-based approach to convert dense language models into sparsely activated ones by initializing router weights from language models.
Outcome:	The proposed architecture improves model capabilities and routing consistency across multiple benchmarks.

Copied to clipboard

Jaeseong Lee, Seung-won Hwang, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He

HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive Learning (2025.findings-acl)

Challenge:	Mixture-of-experts (MoEs) have been adopted for reducing inference costs by sparsely activating experts in large language models (LLMs).
Approach:	They propose a structured-then-unstructured approach outperforming both of structured and unstructured pruning for MoEs.
Outcome:	The proposed approach outperforms both of structured and unstructured pruning, especially for MoEs with hundreds of experts.

Copied to clipboard

Hoang Tran Vuong, Tue Le, Tu Vu, Tung Nguyen, Linh Ngo Van, Sang Dinh, Thien Huu Nguyen

Information Extraction & Text Mining

Challenge:	Recent advances in neural topic models (NTMs) have improved topic quality but still face challenges: weak document-topic alignment, high inference costs due to large pretrained language models, and limited modeling of hierarchical topic structures.
Approach:	They propose a framework that integrates hierarchical clustering and contrastive learning to refine document-topic relationships using compact PLM-based embeddings.
Outcome:	The proposed framework improves topic coherence, topic performance, representation quality and computational efficiency over existing NTMs.

PISCO: Pretty Simple Compression for Retrieval-Augmented Generation (2025.findings-acl)

Copied to clipboard

Maxime Louis, Hervé Déjean, Stéphane Clinchant

Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem? (2025.findings-acl)

Challenge:	Document compression methods suffer from accuracy losses and limited context size.
Approach:	They propose a method that achieves a 16x compression rate with minimal accuracy loss . they show that PISCO outperforms existing compression models by 8% in accuracy .
Outcome:	The proposed method outperforms existing compression models by 8% in accuracy.

Copied to clipboard

Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, Linfeng Zhang

Optimizing Chinese Lexical Simplification Across Word Types: A Hybrid Approach (2024.emnlp-main)

Challenge:	Multimodal large language models have shown remarkable performance for cross-modal understanding and generation, yet suffer from severe inference costs.
Approach:	They propose to prune redundant tokens in MLLMs to reduce computation and storage costs.
Outcome:	The proposed method reduces the computational and storage costs of MLLMs by identifying redundant tokens and pruning them.

Copied to clipboard

ZiHao Xiao, Jiefu Gong, Shijin Wang, Wei Song

SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts (2023.emnlp-main)

Challenge:	Expensive large language models outperform small models in simplifying complex content words and Chinese idioms from the dictionary.
Approach:	They propose to use a retrieval-based interpretation augmentation strategy to refine small models to simplify complex content words and Chinese idioms.
Outcome:	The proposed framework outperforms large language models in Chinese Lexical Simplification (CLS) and improves on OOD models.

Copied to clipboard

Joon-Young Choi, Junho Kim, Jun-Hyung Park, Wing-Lam Mok, SangKeun Lee

TwT: Thinking without Tokens by Habitual Reasoning Distillation with Multi-Teachers’ Guidance (2025.findings-emnlp)

Challenge:	Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models.
Approach:	They propose a prompt tuning method that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced by longer soft prompt.
Outcome:	The proposed method outperforms baseline methods while preserving memory usage.

Copied to clipboard

Jingxian Xu, Mengyu Zhou, Weichang Liu, Hanbing Liu, Shi Han, Dongmei Zhang

C2KD: Cross-layer and Cross-head Knowledge Distillation for Small Language Model-based Recommendation (2025.findings-acl)

Challenge:	Large Language Models (LLMs) have made significant strides in problem-solving by incorporating reasoning processes, but this enhanced reasoning capability results in an increased number of output tokens during inference, leading to higher computational costs.
Approach:	They propose a method that internalizes explicit reasoning into the model’s habitual behavior through a Teacher-Guided compression strategy inspired by human cognition.
Outcome:	The proposed method reduces inference-time costs while maintaining high performance while preserving high quality and diversity of the distillation dataset.

Copied to clipboard

Xiao Chen, Changyi Ma, Wenqi Fan, Zhaoxiang Zhang, Li Qing

Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model (2023.emnlp-main)

Challenge:	Large Language Models (LLMs) show promise but their size and high inference costs limit deployment on resource-constrained devices.
Approach:	They propose a framework to transfer task-relevant knowledge from two complementary dimensions to Large Language Models (LLMs) Large Language models (LLMS) have demonstrated great potential in sequential recommendation tasks .
Outcome:	Extensive experiments across diverse model families show that the proposed framework achieves competitive performance compared to LLMs.

Copied to clipboard

Zeyu Liu, Tim Dettmers, Xi Lin, Veselin Stoyanov, Xian Li

LOHRec: Leveraging Order and Hierarchy in Generative Sequential Recommendation (2025.findings-emnlp)

Challenge:	Large and sparse feed-forward layers (S-FFN) have proven effective in scaling up the model size for pretraining large language models.
Approach:	They compare S-FFN architectures for language modeling and compare their performance and efficiency . they found a simpler selection method that selects blocks through their mean aggregated hidden states .
Outcome:	The proposed model size and selection method achieve lower perplexity in language model pretraining compared to existing MoE architectures.

Copied to clipboard

Jiawen Xie, Haiyang Wu, Deyi Ji, Yuekui Yang, Shaoping Ma

Challenge:	generative recommenders focus on maximizing the prediction probability of the next item in the temporal sequence, ignoring diverse potential items.
Approach:	They propose a learning framework that leverages order and hierarchy in generative recommendation using quantized identifiers to further explore performance ceiling of lightweight generative recommenders.
Outcome:	The proposed learning framework outperforms strong prior baselines across multiple datasets.

On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures (2025.acl-long)

Copied to clipboard

Minh Duc Bui, Kyung Eun Park, Goran Glavaš, Fabian David Schmidt, Katharina Von Der Wense

SADA: Bridging In-Context Learning and Fine-Tuning via State-Aligned Distillation Adapters (2026.acl-long)

Challenge:	Large Language Models (LLMs) should be able to provide accurate information irrespective of the measurement system at hand .
Approach:	They use newly compiled datasets to test if this is true for seven open-source LLMs.
Outcome:	The proposed model can provide accurate information regardless of the measurement system at hand.

Copied to clipboard

Wenhao Gao, Tianlong Wang, Wei Jia, Linhao Zhang, Aiwei Liu, Miao Fan, Zhou Xiao

MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP (2025.findings-acl)

Challenge:	Prompt-based in-context learning and parameter fine-tuning are dominant paradigms for incorporating external information into large language models, but they incur high inference costs or require expensive retraining.
Approach:	They propose to convert prompts into temporary adapter weights to bridge this gap by converting prompts to temporary adapters.
Outcome:	The proposed model outperforms baselines on long-context language modeling and downstream NLU and summarization benchmarks while significantly reducing memory footprint and latency.

Copied to clipboard

Kurt Micallef, Claudia Borg

CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling (2025.findings-emnlp)

Challenge:	Large Language Models (LLMs) have shown remarkable performance across various NLP tasks, largely due to their generalisability and ability to perform tasks without additional training.
Approach:	They evaluate the performance of 55 publicly available Large Language Models on Maltese, a low-resource language, using a newly introduced benchmark covering 11 discriminative and generative tasks.
Outcome:	The proposed models perform poorly on discriminative and generative tasks and smaller fine-tuned models perform better across all tasks.

Copied to clipboard

Xinze Wang, Chen Chen, Yinfei Yang, Hong-You Chen, Bowen Zhang, Aditya Pal, Xiangxin Zhu, Xianzhi Du

SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities (2025.findings-acl)

Challenge:	Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs.
Approach:	They propose an alternative training strategy that converts a dense CLIP model into a sparse MoE architecture.
Outcome:	The proposed training strategy outperforms dense models on COCO and Flickr30k benchmarks.

Copied to clipboard

Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, Radha Poovendran

ReTAG: Retrieval-Enhanced, Topic-Augmented Graph-Based Global Sensemaking (2025.findings-emnlp)

Challenge:	Emerging large reasoning models (LRMs) leverage long chain-of-thought (CoT) reasoning to enhance their reasoning capabilities.
Approach:	They conduct a systematic study of LRM safety using human annotations to assess their safety.
Outcome:	The proposed safety measures are compared to state-of-the-art models on strong and wildjailbreak datasets.

Copied to clipboard

Boyoung Kim, Dosung Lee, Sumin An, Jinseong Jeong, Paul Hongsuck Seo

Natural Language Interfaces

Challenge:	a prior graph-based approach to global sensemaking lacks retrieval mechanisms, topic specificity, and incurs high inference costs.
Approach:	They propose a RetrievalEnhanced, Topic-Augmented Graph framework that retrieves relevant summaries from a topic.
Outcome:	The proposed framework improves response quality while significantly reducing inference time compared to the baseline.

Relevant or Random: Can LLMs Truly Perform Analogical Reasoning? (2025.findings-acl)

Copied to clipboard

Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty

Stop When Enough: Adaptive Early-Stopping for Chain-of-Thought Reasoning (2026.acl-long)

Challenge:	Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences.
Approach:	They propose to use self-generated random examples to improve performance on a variety of reasoning tasks by incorporating relevant examples from relevant past experiences.
Outcome:	The proposed methods achieve comparable or even better performance on GSM8K with random biological examples.

Copied to clipboard

Renliang Sun, Wei Cheng, Dawei Li, Haifeng Chen, Wei Wang

Low-Resource NLP

Challenge:	Chain-of-Thought reasoning has driven recent gains of large language models (LLMs) on reasoning-intensive tasks by externalizing intermediate steps.
Approach:	They propose a training-free framework that adaptively determines when to stop reasoning to mitigate overthinking.
Outcome:	The proposed framework reduces token usage by 20-55% while maintaining or improving accuracy compared to standard CoT prompting.

StitchLLM: Serving LLMs, One Block at a Time (2025.acl-long)

Copied to clipboard

Bodun Hu, Shuozhe Li, Saurabh Agarwal, Myungjin Lee, Akshay Jajoo, Jiamin Li, Le Xu, Geon-Woo Kim, Donghyun Kim, Hong Xu, Amy Zhang, Aditya Akella

Enhancing RAG Efficiency with Adaptive Context Compression (2025.findings-emnlp)

Challenge:	Existing techniques like distillation and pruning are not efficient for large language models.
Approach:	They propose a dynamic model routing framework that uses a powerful bottom model to process all queries and a lightweight routing mechanism to allocate computational resources appropriately.
Outcome:	The proposed framework improves system throughput while minimizing performance degradation.

Copied to clipboard

Shuyu Guo, Shuo Zhang, Zhaochun Ren

Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems (2025.findings-emnlp)

Challenge:	Existing methods apply fixed compression rates, over-compressing simple queries or under-compressed complex ones.
Approach:	a new framework uses a hierarchical compressor and a context selector to optimize inference efficiency . a framework that dynamically adjusts compression rates based on input complexity optimizes inference without loss of accuracy.
Outcome:	Adaptive Context Compression for RAG outperforms fixed-rate methods on Wikipedia and five QA datasets .

Copied to clipboard

Aakriti Agrawal, Rohith Aralikatti, Anirudh Satheesh, Souradip Chakraborty, Amrit Singh Bedi, Furong Huang

Pretraining Context Compressor for Large Language Models with Embedding-Based Memory (2025.acl-long)

Challenge:	Existing approaches to selecting reliable responses from multiple LLMs often depend on external verifiers, human evaluators, or self-consistency techniques.
Approach:	They propose a calibrated log-likelihood-based selection framework to improve multi-LLM performance.
Outcome:	The proposed method outperforms majority voting and exceeds self-consistency performance when using a large number of model calls.

Copied to clipboard

Yuhong Dai, Jianxun Lian, Yitian Huang, Wei Zhang, Mingyang Zhou, Mingqi Wu, Xing Xie, Hao Liao

Squeezed Attention: Accelerating Long Context Length LLM Inference (2025.acl-long)

Challenge:	Efficient processing of long contexts in large language models is essential for real-world applications such as retrieval-augmented generation and in-context learning.
Approach:	They propose a decoupled compressor-LLM framework that preserves contextual information within condensed embedding representations.
Outcome:	The proposed framework outperforms baseline models in three domains and across eight datasets while adapting to different downstream LLMs.

Copied to clipboard

Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, Amir Gholami

Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs (2025.acl-long)

Challenge:	Emerging Large Language Models require long input context to perform complex tasks.
Approach:	They propose an algorithm to reduce the complexity of attention with respect to the fixed context length.
Outcome:	The proposed method reduces the complexity of attention from linear to logarithmic with respect to the fixed context length.

Copied to clipboard

Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Shenlixing Shenlixing, Chenzhan Chenzhan, Xipeng Qiu, Qi Zhang, Tao Gui

Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback (2026.findings-acl)

Challenge:	Multi-head Latent Attention (MLA) is an innovative architecture designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector.
Approach:	They propose a data-efficient fine-tuning method for transitioning from MHA to MLA using a latent vector cache.
Outcome:	The proposed architecture reduces the KV cache size of Llama2-7B by 92.19%, with only 1% drop in LongBench performance.

Copied to clipboard

Jiaye Lin, Mengdi Li, Xufeng Zhao, Wenhao Lu, Peilin Zhao, Stefan Wermter, Di Wang

Challenge:	Existing approaches to align large language models with human preferences are limited in generalizability due to distribution shift, preference label noise, and mismatch of challenging samples with model capacity.
Approach:	They propose a framework that constructs preference pairs with varying difficulty levels and then produces a specific curriculum for reward model training.
Outcome:	The proposed framework improves generalizability of reward models by a significant margin without incurring additional inference costs compared to existing non-curriculum baselines.

BoundRL: Efficient Token-level Structured Text Segmentation through Reinforced Boundary Generation (2026.findings-acl)

Copied to clipboard

Haoyuan Li, Zhengyuan Shen, Sullam Jeoung, Yueyan Chen, Jiayu Li, Qi Zhu, Shuai Wang, Vassilis N. Ioannidis, Huzefa Rangwala

AOT*: Efficient Synthesis Planning via LLM-Empowered AND-OR Tree Search (2026.findings-acl)

Challenge:	Structured texts often contain elements beyond plain language, such as code snippets, which conventional sentence-level segmentation methods cannot handle effectively.
Approach:	They propose a token-level approach that performs efficient token-based text segmentation and label prediction for long structured texts.
Outcome:	The proposed approach outperforms existing models on short-shot prompts and SFT and standard RLVR models on complex LLM prompts.

Copied to clipboard

Xiaozhuang Song, Xuanhao Pan, Xinjian Zhao, Hangting Ye, Shufei Zhang, Jian Tang, Tianshu Yu

Think How to Think: Mitigating Overthinking with Autonomous Difficulty Cognition in Large Reasoning Models (2026.acl-long)

Challenge:	Experimental evaluation shows that AOT* achieves competitive solve rates using 3-5 fewer iterations than existing LLM-based approaches.
Approach:	They propose a framework that integrates LLM-generated chemical synthesis pathways with systematic AND-OR tree search.
Outcome:	Experimental results show that AOT* improves search efficiency and solves faster than existing approaches.

Copied to clipboard

Yongjiang Liu, Haoxi Li, Xiaosong Ma, Jie Zhang, Song Guo

Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction (2026.findings-acl)

Challenge:	Recent Large Reasoning Models (LRMs) excel at complex reasoning tasks but often suffer from overthinking.
Approach:	They propose a two-stage fine-tuning strategy that progressively inspires LRMs’ difficulty cognition and redundancy cognition of LRM.
Outcome:	The proposed model significantly reduces inference costs by over 70% on easy tasks and 40% on complex ones without compromising performance.

Copied to clipboard

Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Yilun Zhu, Shervin Malmasi

Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching (2026.acl-long)

Challenge:	Existing methods for text generation require multiple independent sequences to be decoded in parallel.
Approach:	They propose an algorithm that accelerates offline decoding by leveraging shared memory and computation across batches.
Outcome:	Experiments show that attribute-value pairs are conditionally independent, enabling decoding in parallel up to 96 tokens per prompt.

Copied to clipboard

Bo Lv, Jingbo Sun, Jianwei Lv, Chen Tang, Shaojie Zhang, Nayu Liu, Guoxin Yu, Zihao Li, Qichao Zhang, Dongbin Zhao, Ping Luo, Yue Yu

Steering LLM Thinking with Budget Guidance (2026.findings-acl)

Challenge:	Existing routing methods rely on direct mapping from queries to models based on surface-level features, leading to poor generalizability on out-of-distribution data.
Approach:	They propose a new routing framework that recasts the routing task as a matching process of sifting similar queries from historical logs.
Outcome:	The proposed framework improves matching accuracy while lowering inference costs . it decouples linguistic surface forms from task-intrinsic requirements .

Copied to clipboard

Junyan Li, Wenshuo Zhao, Yang Zhang, Chuang Gan